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Symbols & Abbreviations 


ADAHRS 

ADS-B 

AFFTC 

AGL 

AMM 

ANOVA 

ATC 

ATIS 

ATOL 

CDAP 

CDI 

CDTI 

Con Ops 

DAS 

EMA 

EP 

ETA 

ETE 

F 

FAA 

FAF 

FPM 

FSIL 

ft. 

GA 

GPS 

HSI 

HVO 

Hz 


Air Data, Attitude and Heading Reference System 

Automatic Dependent Surveillance - Broadcast 

Air Force Flight Test Center 

Above Ground Level 

Airport Management Module 

Analysis of Variance 

Air Traffic Control 

Automated Traffic Information System 

Air Traffic Operations Lab 

Conflict Detection, Alerting and Prevention 

Course Deviation Indicator 

Cockpit Display of Traffic Information 

Concept of Operations 

Data Acquisition System 

Exponential Moving Average 

Evaluation Pilot 

Estimated Time of Arrival 

Estimated Time Enroute 

F-ratio 

Federal Aviation Administration 
Final Approach Fix 
Feet Per Minute 

Flight Systems Integration Laboratory 
feet 

General Aviation 
Global Positioning System 
Horizontal Situation Indicator 
Higher Volume Operations 
Hertz 


IAF 

IAP 

IF 

IFR 

IMC 

KIAS 

kts 

LaRC 

LCD 

LVDS 

M 

MAP 

MAS 

MCH 

MFD 

N 

NAS 

NASA 

NCAM 


Initial Approach Fix 

Instrument Approach Procedure 

Intermediate Fix 

Instrument Flight Rules 

Instrument Meteorological Conditions 

Knots Indicated Airspeed 

knots 

Langley Research Center 
Liquid Crystal Display 
Low-Voltage Differential Serial 
Mean (arithmetic average) 

Missed Approach Point 
Method of Approach Separation 
Modified Cooper-Harper 
Multi-Function Display 
Sample size 

National Airspace System 

National Aeronautics and Space Administration 

National Consortium for Aviation Mobility 



NLR 

National Aerospace Laboratory 

n. m. 

nautical miles 

P 

Probability; level of (statistical) significance 

PIC 

Pilot In Command 

PTS 

Practical Test Standards 

ReTA 

Requested Time of Arrival 

RMSE 

Root Mean Squared Error 

SATS 

Small Aircraft Transportation System 

SCA 

Self Controlled Area 

SD 

Standard Deviation 

SP 

Safety Pilot 

sps 

samples per second 

sss 

Self-Separation and Sequencing 

TAA 

Terminal Arrival Area 

TMX 

Traffic Manager software 

X 2 

Chi-square statistic 
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Introduction 


The Small Aircraft Transportation System (SATS) Higher Volume Operations (HVO) Concept of 
Operations (Con Ops) proposes to dramatically increase operating capacity during Instrument 
Meteorological Conditions (IMC) at non-towered, non-radar equipped airports (i.e., “SATS candidate 
airports”) by enabling simultaneous operations of multiple aircraft [1], Currently, National Airspace 
System (NAS) capacity is limited at these airports during poor weather because Air Traffic Control 
(ATC) procedural separation operations allow only one aircraft to fly either an approach or departure 
procedure at a time (i.e., single takeoff/departure or single approach/arrival) thus reducing airport 
capacity. 


The National Aeronautics and Space Administration (NASA) developed SATS HVO procedures and 
Cockpit Display of Traffic Information (CDTI) formats for pilots to use for self-separation, thereby 
allowing multiple aircraft to operate simultaneously resulting in increased capacity at non-towered, non- 
radar equipped aiiports. A Self-Separation and Sequencing (SSS) Flight Experiment was conducted to 
determine the overall viability of these procedures and display formats. During this flight experiment, 
general aviation (GA) pilots hand-flew their aircraft while using a CDTI and the SATS HVO procedures 
to perform self-separation and sequencing tasks. Quantitative and qualitative data were collected to 
determine if there were any detrimental effects on pilots’ perceived workload levels and abilities to fly an 
instrument approach. 


This report describes the test set up and results of the SSS Flight Experiment. 

Background 

Problem Statement 


Although a capacity plateau has been reached within the United States for the commercial air 
transportation system and the NAS, demand for air transportation services continues to increase. An 
approach to increasing total air transportation system capacity and throughput is to enhance access to 
more than 5,000 smaller airports located within the United States. The majority of these smaller airports 
have no control towers and lie outside ATC radar coverage. However, such airports have the potential to 
provide convenient access and service to communities across the country [2], 

SATS Solution 


NASA, partnered with the Federal Aviation Administration (FAA) and the National Consortium for 
Aviation Mobility (NCAM) (i.e., a consortium of U.S. industries, local and state governments, and 
research institutions including universities), is leading a research and development program focused on 
maturing technologies needed for SATS [3]. The long-term goal of SATS is to facilitate equitable, on- 
demand, widely distributed access to more communities in less time [3] . The near-term, five-year goal of 
the SATS Project is to “develop key airborne technologies [and procedures] that permit small aircraft 
operations during near all-weather conditions at and to virtually any touchdown zone at thousands of 
landing facilities (including small airports) in the United States” [2], 
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SATS HVO 


The SATS Project’s initial focus is to prove that four new operating capabilities will enable safe and 
affordable access to virtually any runway in the nation during most weather conditions. The four SATS 
Project objectives (or sub-elements) center on enabling operational capabilities that enhance operational 
efficiency in the current NAS environment. These objectives include: 

• HVO at Non-Towered, Non-Radar Airports; 

• Lower Landing Minimums at Minimally Equipped Landing Facilities; 

• Increased Single -Pilot Crew Safety and Mission Reliability; and 

• En Route Procedures and Systems for Integrated Fleet Operations. 


The overall goal of the SATS HVO sub-element is to increase capacity by enabling the simultaneous 
operation of multiple aircraft in non-radar airspace at and around airports without air traffic control 
towers in nearly all-weather conditions. Two fundamental aspects of the SATS HVO Con Ops include a 
Self Controlled Area (SCA) and an Airport Management Module (AMM) [1], The AMM is an automated 
ground module located at a SATS airport that will provide information regarding SCA status (i.e., active 
or inactive), as well as sequence number information to arriving aircraft so that pilots can sequence 
themselves onto the approach. The SCA is airspace that is established at a SATS airport (i.e., a non- 
towered, non-radar equipped airport) during 1MC. Within the SCA, pilots are responsible for self- 
separation from other aircraft and for sequencing themselves onto the approach. Automatic Dependent 
Surveillance - Broadcast (ADS-B ) is required by participating aircraft and along with a CDTI will enable 
self-separation. This self-separation capability and related procedures will allow the throughput 
associated with these types of aiiports to increase by reducing the need for ATC’s procedural separation. 
Flight within the SCA will be governed by a set of rules and procedures rather than by the AMM. 


Two primary conditions necessary for the success of the SATS HVO Con Ops are: 

• Pilots must be able to sequence themselves for an instrument approach; and 

• Pilots must be able self-separate before and during the approach. 


Current Study 


The SSS Flight Experiment was conducted to collect quantitative and qualitative data to determine if 
GA pilots, asked to hand fly an aircraft while using a CDTI and related SATS HVO procedures, could 
perform self-separation and sequencing tasks without experiencing detrimental effects on their perceived 
workload levels and abilities to fly an instrument approach. The results of this experiment address the 
fundamental question regarding the overall viability of the SATS HVO Con Ops of simultaneous 
operations of multiple aircraft during 1MC at non-towered, non-radar equipped airports [1], 
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Research Objectives 


The first objective of the SSS Flight Experiment was to determine if GA pilots can use a multi- 
function display (MFD) with traffic information, also referred to as a CDTI, to self-separate and sequence 
their ownship aircraft, while following an aircraft, into a non-towered, non-radar equipped airport during 
1MC. To answer this question, six GA pilots were asked to hand fly an aircraft according to the SATS 
HVO procedures defined for straight-in, in -trail approaches as well as “simultaneous arrival” approaches 
(i.e., approaches requiring aircraft to merge). The overall viability of the self-separation and sequencing 
tasks were evaluated in terms of percentage of time that separation and appropriate landing sequence were 
maintained. 


The second objective of the SSS Flight Experiment was to assess how pilots’ workload and abilities 
to fly an aircraft are affected when they use a CDTI to self-separate and sequence their ownship aircraft 
into a non-towered, non-radar equipped aiiport during IMC. Subjective measures of workload and 
objective measures of flight path deviation were recorded while pilots performed the SATS HVO 
approach procedures. Pilots’ workload and flying proficiency levels were also measured when they 
performed straight-in, in-trail approaches and simultaneous arrival approaches using current day FAA 
GPS instrument approach procedures (i.e., without the use of SATS HVO procedures and a related 
CDTI), so that a baseline of each pilot’s flying proficiency could be established. 

Method 

Subjects 


Six GA EPs participated as test subjects. All EPs were male, ranged in age between 19 - 46 years 
[Mean (M) = 24, Standard Deviation (SD) = 10], were instrument-rated and current to fly under 
Instrument Flight Rules (IFR), and held a high performance aircraft endorsement. Two of the EPs held 
private pilot certificates; four of the EPs held commercial pilot certificates; and four of the EPs held 
complex aircraft endorsements. 


All EPs had less than 350 total flight hours (M = 276, SD = 52). On average, the EPs had flown 
approximately 29 hours during the last 90 days (M = 28.8, SD = 21.4), with four of the six EPs having 
flown at least 10 hours while using a global positioning system (GPS) device during the last 90 days. 
Four of the EPs had previous experience flying GPS instrument approaches, and five of the EPs had 
previous experience using a Horizontal Situation Indicator (HSI) flight instrument during flight. 


Prior to participating in the SSS Flight Experiment, only one of the EPs had ever used a CDTI. None 
of the EPs were certified flight instructors, had previously flown a Cirrus SR22 aircraft, or had flown for 
the military. All EPs were treated in accordance with the “Ethical Principles of Psychologists and Code of 
Conduct” [4], 
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Test Facilities and Apparatus 


Facilities and apparatus for the SSS Flight Experiment included the test aircraft; airborne research 
software; and ground simulation software. These items are described in the following sections. 

Test Aircraft 


The test aircraft used for this experiment was NASA Langley Research Center’s (LaRC) Cirrus 
SR22X research aircraft. The SR22X (Figure 1) is a four-place, composite, fixed-gear aircraft with a 
single 310-horsepower piston engine. For this experiment, EPs flew the assigned experiment tasks from 
the left-side pilot’s seat, while a NASA Safety Pilot (SP) flew at all other times from the right seat. An 
experimenter occupied the right aft seat and operated experiment equipment located in the aft area of the 
aircraft (Figure 2). 



Figure 1. NASA LaRC’s Cirrus SR22X research aircraft. 
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Evaluation Pilot 


NASA Safety Pilot 



Experimenter 


Figure 2. SR22X seating arrangement. 


The CitTus SR22X is one of several new-generation GA aircraft making use of the latest in materials, 
aerodynamics, avionics, and manufacturing technology [5]. NASA LaRC’s Cirrus SR22X was modified 
by the addition of a GA baseline research system that included an additional power system to power the 
research systems; an ADAHRS; an Avidyne® MFD; an experimenter workstation; a Data Acquisition 
System (DAS); a sensor system and air data boom; two general-purpose computers; a video system; and 
an audio system [5]. Much of this equipment was mounted on the research equipment pallet located in 
the aircraft’s aft compartment. For the SSS Flight Experiment, specific modifications to the GA baseline 
research system included the addition of experiment-specific software on the general-purpose computers, 
an additional audio channel for experimenter comments, and custom Jeppesen database cards for the 
Garmin 430 GPS. A detailed list of the data recorded by the DAS is included in Appendix A. 


Multi-Function Display. The Avidyne® FlightMax® EX5000 MFD, located in the middle of the 
main instrument panel (Figure 3), was used to present the appropriate display information during each of 
the experiment’s approach tasks. The MFD was a 10.4-inch diagonal liquid crystal display (FCD) with 
an 800 x 600 pixel resolution. The bezel panel around the display contained two rotary knobs and 10 
bezel buttons for display and mode control. Outputs from the knobs and bezel buttons were transmitted 
to the general-purpose research computers on an RS-232 serial bus. The unit also accepted input display 
video from a low-voltage differential serial (FVDS) data bus from the research system to display research 
images on the MFD. 
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Figure 3. SR22X instrument panel. 

Horizontal Situation Indicator Display. A Sandel 3800 HSI was configured throughout the 
experiment as shown in Figure 4. The HSI's navigation source was the number one Garmin 430 GPS. 



Figure 4. HSI configuration. 


Airborne Research Software 

The airborne research software served as a main component of the flight experiment in that it: 

• Generated each approach scenario flown by the EP; 

• Generated a simulated traffic aircraft for the EP to follow; 

• Performed calculations and generated displays to enable the self-separation and self-sequencing 
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tasks; 

• Generated the displays on the MFD; 

• Detected potential conflicts between the SR22X and the simulated traffic aircraft; 

• Generated visual and audio alerts associated with traffic conflicts; 

• Generated visual information intended to assist the EP in the prevention of traffic conflicts; 

• Logged scenario start time, scenario stop time, and information regarding loss of separation 
(lateral and vertical) between the SR22X and the simulated traffic aircraft; 

• Corrected known erroneous Air Data, Attitude and Heading Reference System (ADAHRS) pitot 
static data (see Appendix B); and 

• Flagged the MFD, if there was a loss of flight sensor data. 

The majority of the airborne research software interface checks were performed on the ground using 
X-Plane® flight simulation software [6]. In-flight verification and validation of the airborne research 
software was completed during several pre -experiment, checkout flights. 


Traffic Generator. The Netherlands’ National Aerospace Laboratory’s (NLR) traffic manager 
software (TMX) provided traffic generation and conflict detection, alerting, and prevention (CDAP) 
capabilities, and it also served as the main basis of the airborne research software utilized by the SSS 
Flight Experiment [7]. Additional software was developed for the interface between TMX and the MFD 
and ADAHRS [8, 9]. TMX was used to design and test flight scenarios, with both traffic and simulated 
ownship aircraft, before flight. The flight profile and performance of the traffic aircraft was specified for 
each flight scenario. The traffic aircraft’s flight profile included its start position, heading, altitude, 
calibrated airspeed, and all subsequent waypoints, altitudes, and airspeeds. The same winds aloft 
conditions that the ownship aircraft encountered were used to compute the performance of the traffic 
aircraft. The beginning of each scenario started when the EP flew within a specified lateral and vertical 
distance of a predefined start waypoint. TMX provided consistent initial conditions between the traffic 
aircraft and ownship for each flight scenario among all EPs. 


Research Software. Two critical enhancements were made to the TMX software (hereafter called 
research software) to enable SATS HVO operations for sequencing and self-separation. The first main 
enhancement to the research software was the addition of the capability to initialize both the ownship and 
traffic aircraft with a stalling sequence number [10]. The sequence number identified the lead aircraft and 
the following aircraft. In all test conditions, the ownship aircraft was to follow the traffic aircraft on a 
GPS instrument approach into the destination airport. The other main enhancement to the research 
software enabled self-separation operations during the flight experiment [10]. This enhancement 
consisted of two principle components: a Requested Time of Arrival (ReTA) calculation, and a 
Proceed/Hold calculation. The ReTA was an internal calculation, not presented to the pilot, that 
determined the earliest clock time after which the ownship aircraft could depart the Intermediate Fix (IF) 
inbound on the approach and be reasonably assured of proper lateral separation from the traffic aircraft 
throughout the approach. The minimum lateral separation required throughout the approach was 3 
nautical miles (n. m.). 
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The research software calculated ReTA based on the speed profiles of the traffic aircraft and the 
ownship aircraft. The traffic and ownship aircraft had two speeds that were planned to be flown during 
the approach [i.e., a constant airspeed up to the Final Approach Fix (FAF), followed by a speed reduction 
to final approach speed to the runway threshold]. Once the lead aircraft arrived within specified heading 
and distance values of the IF inbound, the research software, using the actual winds aloft derived onboard 
the ownship aircraft and the planned speed profile of the lead aircraft, calculated the lead aircraft’s 
estimated time of arrival (ETA) at the runway threshold. Then, based on the planned speed profile of the 
ownship aircraft and derived winds aloft, the software calculated the estimated time enroute (ETE) of the 
ownship aircraft from the IF to the minimum lateral separation point plus a 0.5 n. m. margin (i.e., 3.5 n. 
m. from the runway threshold). This ETE was subtracted from the lead aircraft’s ETA. Finally, 30 
seconds were added to the last calculated time. This time period was used in addition to the 0.5 n.m. 
margin to create an additional buffer to account for minor pilot and aircraft performance uncertainties 
such as non-steady state performance and flight path parameter deviation allowances within the FAA’s 
Instrument Rating Practical Test Standards (PTS) [1 1]. The resulting time was the ReTA for the ownship 
aircraft. For simplification, during all calculations, an instantaneous speed reduction by all aircraft was 
assumed at the speed reduction point. Thus, in equation form, the ReTA was calculated as follows: 


(1) Requested Time of Arrival (ReTA) = (ETA of simulated traffic aircraft at runway 
threshold) - (ETE of ownship aircraft from IF to 3.5 n. m. from runway threshold) + 30 
seconds 


If the ownship aircraft accelerated or decelerated inbound to the IF, a new ReTA was calculated using 
the new speed. In short, the ReTA, with some margin for error, was calculated to help ensure at least a 
minimum lateral separation of 3 n. m. at the planned closest point of approach (i.e., when the lead aircraft 
crossed the threshold). For a faster ownship aircraft, its ReTA would be later than a slower aircraft to 
ensure the same minimum separation when the lead aircraft crossed the threshold. 


In an attempt to ease the workload associated with pilot mental calculations aloft, a simple tool was 
developed to assist the pilot in determining whether he had enough separation to begin the approach. This 
tool instructed the pilot, via the MFD, to either “Proceed” on the approach with proper separation or to 
“Hold” until proper separation could be guaranteed. Instead of displaying the ReTA to the EP, the 
research software calculated the ETA of the ownship aircraft at the IF. If the ETA was later than the 
ReTA, then the text “Proceed” was displayed to the EP, and the EP could fly over the IF waypoint 
inbound for the instrument approach. If the ETA was earlier than ReTA, then the text “Hold” was 
displayed to the EP, and the EP was required to hold until he received the “Proceed” indication from the 
software. It is important to note that the puipose of the Proceed/Hold tool was not to achieve a specific 
spacing between aircraft, but to help ensure that separation could be maintained throughout the approach. 

Ground Simulation Software 


The research software and the flight hardware, including the general-purpose computers and the 
Avidyne® MFD, were tested together in the NASA LaRC Flight Systems Integration Laboratory (FSIL) 
and, prior to flight, on-board the test airplane. X-Plane®, a commercially available flight simulation 
software, was used to provide parameters normally generated within the aircraft ADAHRS and GPS 
system during flight. These parameters included ground speed, track, heading, latitude, longitude, 
altitude, airspeed, vertical speed, and wind speed and direction. 
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Evaluation Tasks 


EPs evaluated three types of display formats while performing a series of straight in, in-trail and 
simultaneous arrival GPS instrument approach types. Each display type and approach type is described 
below. 

Display Types 


The Avidyne® FlightMax® EX5000 MFD, located in the middle of the main instrument panel 
(Figure 3), was used to present the appropriate display information during each approach task. 


Baseline Display Format. The display format in Figure 5 was used as a baseline to compare the 
effects of other additional information intended to aid the pilot with his self-separation task. 
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Figure 5. Baseline display format. 


The baseline is a head-up format with a full-compass rose showing the magnetic heading of the ownship. 
An airplane symbol, representing ownship, is fixed in the center of the display. A map display, drawn 
with magenta lines between waypoints, shows the flight plan programmed in the GPS navigation system 
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relative to the airplane symbol. A green line drawn from the airplane symbol towards the top of the 
display represents the current ground track of the ownship. A range ring, with numerical value of 
distance from the ownship (numerical value is one-half of the map scale), is drawn around the airplane 
symbol. The magnetic heading and speed of the current winds computed in the research software are 
shown in the upper left-hand portion of the display. 


A standard GPS holding pattern was displayed at the IF (when selected by the pilot during 
simultaneous approaches). The holding pattern turn radius was calculated by taking the planned indicated 
airspeed of the ownship aircraft [i.e., 120 knots indicated airspeed (KIAS)] plus the absolute value of the 
winds aloft. This accounted for the worst-case (widest) turn radius because of the assumed high ground 
speed for the ownship aircraft. The 4 n. m. outbound leg is also depicted on the display. 


MAS 1 Display Format. The MAS 1 display format (Figure 6) has the same format as the baseline 
with the addition of other-traffic symbology and an approach and weather information box. 



Figure 6. MAS 1 display format. 


Other-traffic-aircraft are displayed as a chevron, relative to the ownship position and ground track, 
with a circle representing the minimum lateral separation to be maintained between ownship and the 
traffic aircraft. An aircraft information tag is attached to the chevron (Figure 7). 
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110-012 V 

PA28 N6664N 


Figure 7. Traffic aircraft information. 


The top line indicates the sequence of the particular traffic aircraft on the instrument approach for 
landing. In this example, “#1” indicates that the aircraft is the lead aircraft on the instrument approach to 
the airport. The second line indicates the computed calibrated airspeed, difference in altitude of the 
aircraft relative to ownship in hundreds of feet, and whether the traffic aircraft has a rate of climb or 
descent greater than 200 feet per minute (fpm). In this example, the traffic aircraft has a calibrated air 
speed of 110 kts based on calculations within the ownship aircraft from traffic state vector information 
and the winds aloft and atmospheric conditions measured from the ownship aircraft’s systems. The traffic 
aircraft is 1,200 ft. below the ownship aircraft and is descending. The third line indicates the type of 
traffic aircraft and its FAA aircraft registration number. In this example, the aircraft type is a PA28 with 
a registration number of N6664N. 


An approach and weather information box is displayed on the lower left-hand side of the display. 
Figure 8 illustrates the information contained in the first two lines of that box. 


Sequence number: #2 

PROCEED 


Figure 8. Sequence number and “Proceed/Hold” text. 


The first line in the top of the approach and weather information box shows the sequence number of 
ownship on the instrument approach to the airport. In this example, the ownship aircraft is the second 
aircraft currently on the approach. When another aircraft on the approach in front of the ownship aircraft 
completes the instrument approach, the sequence numbers cascade forward for all aircraft still conducting 
the approach (in this example, the ownship aircraft’s sequence number would change to #1). The 
ownship aircraft’s sequence number is also shown next to the ownship symbol in the middle of the 
display (Figure 9). 


The CDTI concept used in the SSS Flight Experiment received traffic update information at the same 
one hertz (Hz) update rate that would have been experienced if the test aircraft’s avionics was receiving 
ADS-B information from an actual traffic aircraft. Since the EPs self-separated from simulated traffic 
during simulated IMC, an EP would not have been aware of any presentation differences between the 
display of a simulated traffic aircraft or an actual traffic aircraft. 
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Figure 9. Ownship aircraft and sequence number. 


The second line of the approach and weather information box shows whether the ownship aircraft 
should proceed on with the approach or hold. The proceed indication (shown in green) and the hold 
indication (shown in amber) are computed based on whether proper time -based separation can be 
guaranteed. The remainder of the approach and weather information box shows the name of the 
instrument approach, the name of the initial fix on the instrument approach, the runway number on which 
to land, and the current winds and cloud heights above the airport as reported on the Automatic Terminal 
Information Service (ATIS). 


MAS 2 Display Format. The MAS 2 display format (Figure 10) has the same format as the MAS 1 
display with the addition of CDAP display information. The conflict prevention display information is 
drawn as conflict prevention bands on the airspeed and vertical speed indicators shown on the upper left- 
hand side of the display and on the compass rose. Conflict detection is also shown by color changes 
made to the conflicting traffic symbology (chevron, information tag, and minimum separation ring) and 
by audible tones on the intercom system. 



Figure 10. MAS 2 display format. 
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A conflict, or the potential loss of separation, occurs when the ownship aircraft’s instantaneous flight 
path (state vector based) penetrates the lateral (3 n. m.) or vertical (+/- 1,000 ft.) airspace surrounding a 
traffic aircraft (during its state vector based flight path). Conflict detection is calculating within the next 
30 seconds and the next 30 to 60 seconds, potential losses of separation between the ownship aircraft and 
all traffic aircraft. When a potential loss of separation is detected, the pilot of the ownship aircraft is 
notified with visual indications (i.e., a change in traffic aircraft display color) and audible alerts. Conflict 
prevention is calculating potential losses of separation with traffic aircraft based on a single parameter 
(e.g., airspeed, altitude, or heading). Conflict prevention bands indicate to the pilot, on the airspeed 
indicator, altimeter, or compass rose, which values to avoid to remain separated (i.e., the pilot should not 
fly into the indicated bands). When the prediction logic indicates a loss of separation within the next 30 
to 60 seconds, an alert with cyan conflict prevention bands on either the airspeed or vertical speed 
indicator shows the pilot of the ownship aircraft the airspeed range or vertical speed range to avoid if loss 
of separation is to be prevented (Figures 1 1 and 12). 



Figure 11. Vertical speed conflict prevention band. 



Figure 12. Airspeed conflict prevention bands. 


If loss of separation is predicted to occur in less than 30 seconds, the conflict prevention bands are amber 
colored. If the ownship aircraft were to penetrate the minimum lateral separation from the other traffic, 
an audible tone would sound, and all conflict prevention bands, including a heading band, would appear 
on the display to assist the pilot with regaining separation. 
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Approach Types 

Straight-In, In-Trail Approach. Figure 13 depicts a typical scenario, including level-flight and step 
down procedures, used during the SSS Flight Experiment. During a straight-in, in-trail approach, the EP 
would proceed direct to the IF and fly the approach without any major changes in heading. The EP’s task 
involved maintaining at least the minimum lateral separation while following the lead aircraft on the 
approach. As mentioned previously, the straight-in, in-trail scenarios were designed so that when the 
traffic aircraft landed, the ownship aircraft was 3.5 n. m. plus 30 seconds from the runway threshold. 


IAF 



Figure 13. Straight in, in-trail approach diagram. 

Simultaneous Arrival Approach. Figure 14 depicts a flight scenario that required EPs to fly a 
holding pattern before beginning the approach. During a simultaneous arrival, the flight scenarios were 
designed so that the ownship aircraft would arrive at the IF at nearly the same time as the lead aircraft. 
This forced the EP to maintain a vertical separation from the traffic aircraft and forced him to fly a 
holding pattern until receiving the “Proceed” message. EPs were required to complete the full hold so 
that SATS HVO simultaneous arrival approaches could be compared to the baseline simultaneous arrival 
approach. In actual implementation, a pilot could shorten the hold if given the “Proceed” indication early 
in the hold (as the EPs were given in the current experiment). 
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Figure 14. Simultaneous arrival approach diagram. 

The self-separation task for both types of approaches ended when the ownship aircraft received the 
indication of having “Sequence #1” for the airport. This message was received after the lead aircraft had 
cleared the runway and was no longer a factor. 

Test Areas 


To minimize the interference of non-participating aircraft with data runs, the data runs were 
performed at a minimum of 2,000 ft. above ground level (AGL), using virtual GPS instrument 
approaches, instead of into an airfield. To maximize flexibility with winds aloft, approaches to opposite 
ends of two perpendicular runways were developed (for a total of four different approaches). The flight 
experiment’s GPS instrument approaches were developed using the FAA’s standard terminal arrival area 
(TAA) criteria as an initial starting point (Figure 15) [12]. 



Figure 15. FAA’s standard GPS “T”. 


Depending on the wind direction and presence of non-participating traffic on a particular flight day, one 
of the four different GPS instrument approaches was selected by the onboard experimenter. 
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Each GPS approach’s waypoints were determined using Jeppesen’s FliteMap IFR North America® 
software [13]. To support the self-separation task, a mandatory holding pattern, with the inbound course 
aligned with the final approach course, was established at the straight-in initial approach fix IF(IAF) for 
each runway. Flying direct to the IF allowed an arriving pilot to better anticipate the actions of other 
aircraft (this task would have been more difficult if all traffic aircraft had instead intercepted the final 
approach course at a different point). To minimize airspace used by the flight experiment, some of the 
opposing runway approaches were designed to utilize the same waypoint. 

Approach Charts and Procedures 


An instrument approach procedures (LAP) chart was developed in-house for each of the four different 
approaches flown during this flight test. These charts, as shown in Figure 16, are similar in format to 
standard GPS IAP charts published by the FAA. The charts show the pilot the lateral path to be flown 
from one of four initial approach fixes (IAF) to the runway and then a path to a holding waypoint if a 
landing cannot be completed. The minimum altitude that a pilot can be at when passing the waypoints 
and the minimum descent altitude and minimum visibility necessary for the pilot to complete the landing 
are also shown. 


The SATS HVO instrument approach procedures utilized the same design criteria established by the 
FAA for normal GPS approaches except for modifications made to accommodate multiple aircraft 
simultaneously. These modifications included a holding pattern depicted at the IAF at the beginning of 
the final course in-bound to the runway 1,000 feet (ft.) above the minimum crossing altitude for that IAF. 
The SATS HVO procedures required aircraft that had no conflicting traffic between them and the runway 
to cross the IF at the minimum crossing altitude (4,500 ft. in Figure 16) and proceed with the approach 
without entering the holding pattern. If there was conflicting traffic, a “HOFD” message would be shown 
on the MFD, and the trailing aircraft would be required to hold 1,000 ft. above the preceding aircraft’s IF 
minimum crossing altitude (5,500 ft. in Figure 16). When the preceding aircraft was no longer a conflict, 
a “PROCEED” message would be shown on the MFD, and the holding-aircraft could descend and start 
the approach. The “PROCEED/HOFD” messages were computed by the research software to assist the 
EPs in choosing the appropriate approach segment to fly (i.e., holding procedure or final approach 
course). 
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NASAVILLE FIELD (NASA) 
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Figure 16. One of four typical SATS HVO approach charts. 


For ease of implementation on the test aircraft, custom Approach Database cards containing SATS 
HVO approach database information were developed in conjunction with Jeppesen Sanderson, Inc. for 
use in the standard dual Garmin GNS 430 GPS/communication/navigation units installed onboard the 
SR22X. These cards enabled the EPs to fly the SATS HVO approaches using the SR22X’s standard 
instrumentation with little additional training. When considering the airborne equipment, it is important 
to note that even though approaches were made to virtual airports, the responses of the avionics (as flown 
by EPs) were the same as if the approaches had been to actual airports. 
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Procedures Used to Develop Experiment Flight Scenarios. A unique flight scenario was 
developed for each of the four runways used during the SSS Flight Experiment. For each runway, a flight 
scenario was created to capture: 1) the courses of both IAFs to the IF, 2) approach type (i.e., straight-in, 
in-trail or simultaneous arrival with simulated traffic), 3) display type [i.e., Baseline, Method of Approach 
Separation (MAS) 1, or MAS 2], and 4) four wind variation considerations [i.e., 5, 15, 25, and 35 knots 
(kts)]. Initial conditions for position, heading, altitude, and airspeed for the traffic aircraft associated with 
each flight scenario were determined as well. As a result, a flight scenario database consisting of 192 
flight configuration files (i.e., [{(4 winds aloft conditions x 6 test conditions) x 2 inbound courses from 
the IAFs} x 4 runways]) was developed. This large number of scenario options allowed the onboard 
experimenter to account for true environmental conditions occurring during flight, thus producing the 
highest quality data for post-test processing. 


The initial conditions for each of the flight scenarios (i.e., the start points for the ownship aircraft and 
the traffic aircraft) were defined such that when the lead (i.e., traffic) aircraft crossed the runway 
threshold, the ownship aircraft would always be located at the same distance in-trail behind the lead 
aircraft. Defining each flight scenario’s initial conditions was accomplished using the Jeppesen 
FlightMap and the research software applications [10, 13]. 


A set of computer scenarios was developed for test in the NASA FaRC Air Traffic Operations Fab 
(ATOF), and then the same set of flight scenarios was designed to be loaded directly onto the test 
aircraft’s experimental systems equipment. Each scenario was flown and tested in the ATOF to ensure its 
quality and correctness before being inserted into the SR22X data system. Additionally, pre-test flights 
were performed to verify software configurations and display functionality. 

Experiment Design 


The experiment design used for data collection was a 2 [Approach Type (Straight In, In-trail and 
Simultaneous Arrival)] x 3 [Display Type (Baseline, Method of Approach Separation 1, and Method of 
Approach Separation 2)] full-factorial, within-subject design (Figure 17). Six EPs performed all six test 
conditions twice in partially counterbalanced order under simulated IMC using Foggles. Test conditions 
were presented in partially counterbalanced order (i.e., each test condition preceded and followed every 
other test condition and was presented an equal number of times) to control for ordering effects. Since 
each test condition was completed twice, the term “replicate #1” is used to describe the first time that a 
test condition was performed, and the term “replicate #2” is used to describe the second time that a test 
condition was performed. 
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DISPLAY TYPE 


Baseline 

Evaluation Pilots 
1-6 

Evaluation Pilots 
1-6 

Method of Approach 

Evaluation Pilots 

Evaluation Pilots 

Separation (MAS) 1 

1-6 

1-6 

Method of Approach 

Evaluation Pilots 

Evaluation Pilots 

Separation (MAS) 2 

1-6 

1-6 


Straight In, 
In-trail 


Simultaneous 

Arrival 


APPROACH TYPE 

Figure 17. Experiment design matrix. 

Detailed information regarding the method used to develop flight scenarios associated with 
combinations of display type and approach type as well as combinations of runways and winds aloft is 
provided above in the “Approach Charts and Procedures” section. Although 72 data runs were performed 
during this experiment, only 48 of those runs used the SATS HVO display formats and procedures and 
had traffic for the EP to follow. The other 24 runs involved procedural separation and use of the baseline 
display format. Data runs involving the use of the baseline display format and current day approach 
procedures were used to compare pilot tracking performance and workload with the runs using the SATS 
HVO display formats and procedures. 

Independent Variables 


The two independent variables used in the experiment design were display type and approach type. 
Dependent Measures 


Separation Breaches and Landing Sequence Blunders. For each scenario involving SATS HVO 
procedures, EPs were instructed to maintain at least 3 n. m. of lateral separation or 1,000 ft. of vertical 
separation between their ownship aircraft and traffic aircraft and were instructed to maintain landing 
sequence #2 at all times. 

The employment of separation standards for the SATS HVO SSS Flight Experiment was based on 
those standards that ATC has established for aircraft operating on IFR flight plans, below 18,000 ft. 
within a radar environment. When aircraft were at the same altitude, a minimum of 3 n. m. separation 
was required. Three nautical miles were required for either longitudinal (in-trail) or lateral separation. 
Since both values were the same for this experiment, in this document the term “lateral separation” refers 
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to both longitudinal and lateral separation. When aircraft are within 3 n. m. laterally of each other, a 
1,000 ft. minimum vertical separation is required. During the SSS Flight Experiment, EPs were deemed 
to have been vertically separated (within 3 n. m. of traffic) if they remained separated from the traffic 
aircraft by at least 825 ft. This altitude difference occurred as a result of the ±100 ft. allowance in the 
FAA’s Instrument Rating PTS and an allowable ± 75 ft. error in the ownship aircraft altimeter [11], 
There was no altitude allowance for the traffic aircraft (which was simulated) since it “flew” without 
error. 


Frequency counts and durations (i.e., elapsed times) of separation breaches and landing sequence 
blunders were collected to enable the calculation of the percentage of time that separation and appropriate 
landing sequence were maintained. 


Flight Path Parameter Deviation. The EPs were instructed to maintain assigned altitude within 100 
ft. and maintain assigned airspeed within 10 kts while conducting each instrument approach. These 
minima are specified in the FAA’s Instrument Rating PTS [1 1]. The EPs were also instructed to maintain 
a lateral path deviation less than a full-scale deflection on the HSI course deviation indicator (CDI) (± 1.0 
n. m.) while on each approach prior to the IF waypoint and within three-quarter scale deflection (+ 0.225 
n. m.) when past the IF waypoint. 


Subjective Assessments of Workload. Subjective assessments of workload for each test run were 
obtained through the use of the Air Force Flight Test Center’s (AFFTC) Seven-Point Subjective 
Workload Estimate Scale and the Modified Cooper-Harper (MCH) Rating Scale [14, 15]. The AFFTC’s 
subjective workload estimate scale required EPs to integrate contributing factors to workload (activity 
level, system demands, time loads, and safety concerns) and arrive at an overall workload rating ranging 
from “1” (indicating nothing to do; no system demands) to “7” (indicating overloaded; system 
unmanageable; essential tasks undone; unsafe) (Figure 18). 


1) Nothing to do; No system demands. 

2) Light activity; Minimum demands. 

3) Moderate activity; Easily managed; Considerable spare time. 

4) Busy; Challenging but manageable; Adequate time available. 

5) Very busy; Demanding to manage; Barely enough time. 

6) Extremely busy; Very difficult; Non-essential tasks postponed. 

7) Overloaded; System unmanageable; Essential tasks undone; Unsafe. 


Figure 18. Air Force Flight Test Center’s (AFFTC) Seven-Point Subjective Workload Estimate Scale. 

The MCH Rating Scale required EPs to make a series of decisions regarding: whether or not the 
instructed task could be accomplished most of the time; if adequate performance was attainable (i.e., if 
errors were small and inconsequential); and whether or not the level of mental workload required by the 
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instructed task was acceptable. Upon answering questions according to a predetermined logical sequence, 
an overall rating ranging from “1” (indicating that the instructed task was very easy/highly desirable; 
operator mental effort was minimal; and desired performance was easily attainable) to “10” (indicating 
that the instructed task was impossible; it could not be accomplished reliably) was selected (Figure 19). 



Difficulty Level 

Operator Demand Level 

Ratinq 


VERY EASY, 
Highly desirable 

Operator mental effort is minimal and 
desired performance is easily attainable 

1 

-i 

EASY, 

Desirable 

Operator mental effort is low and 
desired performance is attainable 

2 


FAIR, 

Mild Difficulty 

Acceptable operator mental effort is required 
to attain adequate system performance 

3 



MINOR BUT ANNOYING 
DIFFICULTY 

Moderately high operator mental effort is required 
To attain adequate system performance 

4 

MODERATELY OBJECTIONABLE 
DIFFICULTY 

High operator mental effort is required 
To attain adequate system performance 

5 

VERY OBJECTIONABLE BUT 
TOLERABLE DIFFICULTY 

Maximum operator mental effort is required 
To attain adequate system performance 

6 


MAJOR DIFFICULTY 

Maximum operator mental effort is required 
To bring errors to moderate level 

7 

MAJOR DIFFICULTY 

Maximum operator mental effort is required 
To avoid large or numerous errors 

8 

MAJOR DIFFICULTY 

Intense operator mental effort is required 
To accomplish task, but frequent or 
Numerous errors persist 

9 


Major deficiencies, 
System redesign 
Is mandatory. 


M IMPOSSIBLE 

Instructed task cannot be accomplished reliably 

10 



Figure 19. Modified Cooper-Harper (MCH) Rating Scale. 


Procedure 


Each EP individually completed a pre-experiment session, a “classroom” training session, an “in-the- 
aircraft” training session, a familiarization flight, four experiment flights (during which the EP completed 
each of the six test conditions twice), and a post-experiment debriefing session. Each EP participated in 
the experiment over the course of three approximately eight and a half hour days. 


A 15-minute pre -experiment session involved obtaining EPs’ total and recent flight hour data and 
asking EPs to read and sign an informed consent form. During the 3 hour “classroom” training session 
which immediately followed the pre-experiment session, an experimenter presented the EPs with slides 
that provided an overview of: 1) the HVO sub-element of the SATS Project, 2) the purpose of the SSS 
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Flight Experiment, 3) the experiment design matrix, 4) NASA LaRC’s SR22X research aircraft, 4) the 
experiment equipment and procedures, 5) the test areas, 6) the GPS instrument approach tasks, including 
use of the Garmin 430, 7) the interpretation of the HSI, 8) the MFD and its operation, and 9) the 
subjective workload estimate scales. The “classroom” training session also involved having the EPs 
complete simulated approaches (both baseline and SATS HVO) using a simulated Cirrus aircraft, a 
simulated Garmin 430 GPS, and a simulated MFD with traffic information and software produced bezel 
buttons [9]. Components of this desktop simulation are shown in Figure 20. 



Figure 20. Components of the “classroom” training session’s desktop simulation flight. 


An “in-the-aircraft” training session, lasting approximately 30 minutes, immediately followed the 
classroom training session. This session provided review and reinforcement of material covered in the 
classroom session and was conducted in the SR22X. The EPs were given a “hands-on” tour and review 
of the SR22X cockpit instruments, avionics, intercom, flight controls, and MFD and were encouraged to 
ask questions about any unclear items or procedures. 


After the “in-the-aircraft” training session, the EP, an experimenter, and a NASA SP conducted a 
familiarization flight. This flight lasted approximately 2 hours and was used to familiarize the EP with 
the flight characteristics and power settings of the SR22X as well as ah aspects of the tasks he would be 
asked to perform during subsequent experiment flights. After takeoff, departure, climb, and level-off by 
the NASA SP, control of the aircraft was transferred to the EP. The EP was then guided through a series 
of basic flight maneuvers, including straight-and-level flight, standard-rate turns, and 700 fpm descents. 
Following these basic flight maneuvers, the EP was guided through two test conditions involving straight- 
in, in-trail approaches and two test conditions involving simultaneous arrival approaches. The NASA SP 
flew the aircraft during the transitions between test conditions. The four “practice test conditions” and the 
procedures under which they were conducted were designed to progressively expose an EP to ah of the 
component tasks and procedures that would be asked of him during the test runs. Throughout the 
familiarization flight, the EP’s performance was assessed and remedial instruction and commentary was 
provided, as necessary, to assure his full understanding of the expected experiment flight procedures. At 
the conclusion of the familiarization flight, the NASA SP resumed control of the aircraft and landed. This 
concluded the first day of the experiment. 
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During the second and third days of the experiment, four data collection flights were completed. As 
on the familiarization flight, the NASA SP performed the takeoff, climb, and level-off, and then control 
was transferred to the EP. Unlike on the familiarization flight, the EP no longer received explicit 
procedures training for the required tasks and was directed to perform test conditions by the onboard 
experimenter. During each test condition, EPs were instructed to make position reports at the IF, FAF, 
and missed approach point (MAP). When the approaches associated with a given data collection flight 
were completed, the NASA SP again resumed control of the aircraft and landed. A 30-minute debriefing 
session after the final data collection flight completed the experiment’s schedule. 


The experimenter and the NASA SP carried out specific duties during both the familiarization and 
experiment flights. The experimenter selected the appropriate flight scenario and ensured that the 
appropriate information appeared on the CDTI; solicited and recorded the EP’s test condition workload 
ratings; operated the experiment pallet and video recording system; and provided simulated ATC 
instructions to the EP as part of the prescribed experiment flight tasks. The experimenter also directed the 
NASA SP in repositioning the aircraft for each upcoming flight task. The NASA SP performed takeoffs, 
departures, arrivals, and landings; repositioned the aircraft between the EP’s flight tasks; coordinated and 
communicated with ATC; and acted as pilot-in-command (PIC) throughout each flight. 


SP Checklist 


A GPS/HSI checklist was developed for use by the NASA SP so that the Garmin 430 GPS and the 
Sandel 3800 HSI would be consistently configured for every EP. This ensured proper sequencing and 
selection of the correct start waypoints on the Garmin 430 during the approaches and ensured that the 
proper navigation page was displayed to the EPs. 


EP Checklist 


EPs were provided with a checklist for use during the approach procedures to provide consistency 
among the EPs in their performance of the GPS approaches. This checklist specified manifold pressure 
settings, airspeeds, aircraft configuration, communication points, and Garmin 430 interaction at specific 
points during the approach. Power settings at experiment-specific airspeeds and configurations, for 
straight-and-level flight and descents, were determined during sensor checkout flights. 

Results and Discussion 

A flight experiment was conducted to assess whether GA pilots could use a CDTI and related SATS 
HVO procedures to self-separate and sequence their ownship aircraft behind a traffic aircraft at a non- 
towered, non-radar airport during IMC. The effects of two different initial conditions between ownship 
aircraft and traffic as well as the effects of two different CDTI formats compared to a baseline display 
format with no other traffic displayed were investigated. Statistical analyses were performed to examine: 
1) separation breaches and landing sequence blunders, 2) flight path parameter deviations, 3) adherence to 
FAA PTS performance criteria for the Instrument Rating, and 4) subjective assessments of workload. 
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A 5-percent significance level for the statistical analyses of all data collected in this experiment was 
set a priori. Detailed results of the statistical analyses may be found in Appendixes D, E, and F. 

Separation Breaches and Landing Sequence Blunders 


Examining the 48 test runs that had a traffic aircraft included and where the SATS HVO procedures 
were used in conjunction with a CDTI showed that: 1) there were no lateral or in-trail losses of separation 
(< 3 n.m.); 2) there were no vertical losses of separation (< 1,000 ft.); and 3) there were no losses of 
proper landing sequence. The EPs successfully self-separated and sequenced themselves 100-percent of 
the time from a single traffic aircraft. This finding demonstrates that GA pilots could use a CDTI and 
related SATS HVO procedures to self-separate from and sequence their ownship aircraft with one other 
traffic aircraft into a non-towered, non-radar equipped airport during IMC under the given conditions of 
this experiment. 


Data from the 24 test runs involving the use of the baseline display format and current day instrument 
approach procedures were not included in this analysis since other traffic was not displayed to the EPs 
during those runs. 

Flight Path Parameter Deviation 


Deviations from assigned altitude and airspeed and the lateral path deviations from the assigned route 
on the instrument approach were used to quantify flight performance for each of the runs with two 
different initial conditions and three different MFD formats. Root mean squared error (RMSE) values 
were calculated for these parameters for each test run. Airspeed and lateral path deviation data were 
assessed during both level-flight and descents. Altitude data were assessed only during level-flight 
segments. Details of these calculations may be found in Appendix C. 


A series of 2 (Approach Type) x 3 (Display Type) x 2 (Replicate) Analysis of Variance (ANOVA) 
tests was conducted on the RMSE values of altitude, airspeed, and lateral path deviations to determine if 
significant differences existed in these values when a given display type was used during a given 
approach type [16, 17, 18]. Detailed results of the flight path parameter deviation data analyses are 
available in Appendix D. 


Altitude and Lateral Path Deviation 


Analyses of the RMSE values associated with altitude and lateral path deviation revealed that: 


• No statistically significant differences (p > 0.05) were found to exist among the altitude and 
lateral path deviation data associated with approach type; 

• No statistically significant differences (p > 0.05) were found to exist among the altitude and 
lateral path deviation data associated with display type; and 
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• No significant (p > 0.05) Approach Type x Display Type interaction effects were found to 
exist. 

These results indicate that EPs were able to maintain the appropriate altitude and lateral path equally well 
when they performed straight -in, in-trail approaches and simultaneous arrival approaches; when they 
performed approaches using the Baseline display format, the MAS 1 display format, and MAS 2 display 
format; and when they performed different approach types using different display types. Altitude RMSE 
values associated with all six test conditions (Figure 21) reveal means of approximately 40 ft. and 
standard deviations of approximately 13 ft., and lateral path RMSE values associated with all six test 
conditions (Figure 22) reveal means of approximately 0.08 n. m. and standard deviations of 
approximately 0.04 n. m. 


70 , 


60 4 



Straight In Sim. Arrival Straight In Sim. Arrival Straight In Sim. Arrival 
Baseline Baseline MAS 1 MAS 1 MAS 2 MAS 2 


Approach Type x Display Type 

Figure 21. Mean altitude deviations associated with Approach Type x Display Type. 
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Straight In Sim. Arrival Straight In Sim. Arrival Straight In Sim. Arrival 
Baseline Baseline MAS 1 MAS 1 MAS 2 MAS 2 


Approach Type x Display Type 

Figure 22. Mean lateral path deviations associated with Approach Type x Display Type. 

An analysis of the altitude RMSE values associated with each replicate revealed that smaller altitude 
deviations occurred when EPs completed replicate #2 (F [1, 5] = 7.44; p = 0.041). This finding is most 
likely a result of a practice or learning effect since EPs always performed replicate #2 after completing 
replicate #1. 

Airspeed 

EPs were able to maintain the appropriate airspeed equally well when they performed approaches 
using the Baseline display format, the MAS 1 display format, and MAS 2 display format and when they 
performed different approach types using different display types. Airspeed RMSE values associated with 
all six test conditions (Figure 23) reveal means of approximately 3 knots and standard deviations of 
approximately 1 knot. 
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Figure 23. Mean airspeed deviations associated with Approach Type x Display Type. 


At a statistically significant level, EPs were able to maintain airspeeds more accurately when they 
performed simultaneous arrival approaches than when they performed straight-in, in-trail approaches (F 
[1, 5] = 6.80; p = 0.048). Although this finding is statistically significant, a mean airspeed difference of 
just one-quarter of a knot (i.e., 0.25 kt) is operationally insignificant since, even under the best conditions, 
a pilot would be challenged to maintain airspeed within plus or minus 1.0 kt using an aircraft’s standard 
instruments. 


Overall, the analyses of the RMSE values associated with the SSS Flight Experiment’s test conditions 
indicated that flight path parameter deviations did not increase when the SATS HVO self-separation and 
sequencing tasks were added to a baseline GPS instrument approach. EPs flew the SR22X research 
aircraft equally well when they performed baseline GPS instrument approaches and when they performed 
SATS HVO approaches. 

Adherence to PTS for the Instrument Rating 


Percentages of time that EPs failed to adhere to the PTS for the Instrument Rating were also used to 
quantify flight performance. The duration (i.e., elapsed time in seconds) of each failure to fly within the 
PTS was the difference between the point in time when a PTS parameter was breached and the point in 
time when adherence to that PTS parameter was regained. For a given flight path parameter, the total 
time associated with failures to fly within PTS performance criteria was divided by the total time flying to 
obtain a percentage of time spent flying out of PTS conformance. 
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Percentages of time that EPs failed to adhere to the PTS were calculated for altitude, airspeed, and 
lateral path deviation. Data used to assess PTS conformance were obtained from 30-second segments 
during descents where the parameters would be stabilized. Airspeed and lateral path deviation data were 
assessed during both level-flight and descents. Altitude data were assessed only during level-flight 
segments. 


All EPs successfully adhered to the PTS performance criteria for lateral tracking for the Instrument 
Rating (i.e., ± 1.0 n. m. prior to the FAF, and ± 0.225 n. m. after the FAF) 100-percent of the time. 
Therefore, 2 (Approach Type) x 3 (Display Type) x 2 (Replicate) ANOVAs were only conducted on the 
percentages of time that EPs failed to adhere to the PTS for altitude and airspeed to determine if 
significant differences existed in these values when a given display type was used during a given 
approach type [16, 17, 18]. Analyses of the altitude and airspeed data in terms of adherence to PTS 
revealed that, for each flight path parameter, EPs adhered to the corresponding PTS equally well when 
they performed straight-in, in-trail approaches and simultaneous arrival approaches; when they performed 
approaches using the Baseline display format, the MAS 1 display format, and MAS 2 display format; and 
when they performed different approach types using different display types. These findings indicate that 
the EPs’ abilities to fly the SR22X research aircraft during GPS instrument approaches were not affected 
by the addition of self-separation and sequencing tasks. Detailed results of these data analyses are 
available in Appendix E. 


Percentages of altitude PTS nonconformance associated with all six test conditions (Figure 24) reveal 
means of approximately 1.5-percent and standard deviations of approximately 2.8-percent, and 
percentages of airspeed PTS nonconformance associated with all six test conditions (Figure 25) reveal 
means of approximately 0.7-percent and standard deviations of approximately 1.8-percent. (Note that a 
value of zero indicates conformance 100-percent of the time.) 


32 



8 


a> 


TJ 


-H- 

a) 

o 

'■M 

C 

C 

(0 

*— 

E 

o 

V- 

a> 

o 

>4— 

u> 

ro 

E 

o 

c 

o 

a> 

c 

o 

o 

a> 

z 

0. 

CO 

c 

H 

TO 

0. 

a> 


S 



> 

3 _ 

<u 

■*—> 

c 

Q> 

o 

c 

a> 

TJ 


£ 4 


C 

o 

O 


in 

a> 


1 2 


- 

r 





0.75 


2.3 







0/ 

71 



2.65 


1.77 


1.09 


Straight In Sim. Arrival Straight In Sim. Arrival Straight In Sim. Arrival 
Baseline Baseline MAS 1 MAS 1 MAS 2 MAS 2 


Approach Type x Display Type 


Figure 24. Mean percentages of altitude Practical Test Standard (PTS) nonconformance associated with Approach 

Type x Display Type. 
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Straight In Sim. Arrival Straight In Sim. Arrival Straight In Sim. Arrival 
Baseline Baseline MAS 1 MAS 1 MAS 2 MAS 2 


Approach Type x Display Type 

Figure 25. Mean percentages of airspeed Practical Test Standards (PTS) nonconformance associated with Approach 

Type x Display Type. 


Subjective Assessments of Workload 

The AFFTC’s Seven-Point Subjective Workload Estimate Scale and the MCH Rating Scale were 
used by the EPs to rate the level of workload that they experienced during each of the experiment’s six 
unique test conditions [14. 15]. Each of the EPs flew each of the six test conditions twice. An average 
rating for each test condition for each EP was computed. This procedure was used for both rating scales 
with the results used for workload analyses. Nonparametric tests were employed as a conservative 
method for analyzing workload ratings associated with discrete rating scale items. Detailed results of 
these analyses are available in Appendix F. 

The analyses of the AFFTC Subjective Workload Estimate Rating Scale data and the MCFI Workload 
Rating data revealed that EPs reported experiencing equivalent levels of workload when they performed 
straight -in, in-trail approaches and simultaneous arrival approaches; when they performed approaches 
using the Baseline display format, the MAS 1 display format, and MAS 2 display format; and when they 
performed different approach types using different display types. These results suggest that the EPs’ 
perceptions of workload were not affected by the additional tasks of self-separating and sequencing 
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during GPS instrument approaches. 


AFFTC workload ratings associated with all six test conditions (Figure 26) reveal means of 
approximately 2.7 and standard deviations of approximately 0.5, and MCH workload ratings associated 
with all six test conditions (Figure 27) reveal means of approximately 2.3 and standard deviations of 
approximately 1.0. 
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Figure 26. Mean Air Force Flight Test Center (AFFTC) Subjective Workload Estimate Scale ratings associated with 

Approach Type x Display Type. 
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Straight In Sim. Arrival Straight In Sim. Arrival Straight In Sim. Arrival 
Baseline Baseline MAS 1 MAS 1 MAS 2 MAS 2 


Approach Type x Display Type 

Figure 27. Mean Modified Cooper-Harper (MCH) workload ratings associated with Approach Type x Display Type. 

Conclusions 


Two critical pilot tasks integral to the SATS HVO Con Ops include self-separation and sequencing. 
The focus of the SSS Flight Experiment was to determine the feasibility of asking GA pilots to self- 
separate and sequence their ownship aircraft for and during an instrument approach at a non-towered, 
non-radar equipped airport during 1MC. The experiment concentrated on assessing the impact of the self- 
separation and sequencing tasks on EPs’ performance of GPS instrument approaches and perceptions of 
workload during IMC. The additional information displayed to EPs participating in this experiment 
included CDTI, Proceed/Hold messages, and, when using the MAS 2 display format, CDAP information. 

GA pilots successfully used a CDTI and related SATS HVO procedures, to self-separate from and 
sequence their ownship aircraft with a single traffic aircraft into a non-towered, non-radar equipped 
airport during IMC, 100-percent of the time under the conditions of the SSS Flight Experiment. 
Quantitative analyses of the flight path deviation data (altitude, lateral path, and airspeed as well as 
adherence to the PTS performance criteria) suggest that a GA pilot’s ability to fly an instrument approach 
is not adversely affected by the additional tasks of self-separating and sequencing using SATS HVO 
procedures. Furthermore, analyses of qualitative data collected during this experiment indicate that the 
level of workload experienced by a pilot, while flying an instrument approach and performing self- 
separation and sequencing tasks using SATS HVO procedures, is no greater than that experienced when 
performing baseline (i.e., current day) approaches. 
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APPENDIX A. Time-Coded Data Recorded by the Data Acquisition System 
(DAS) 


State variables and lateral flight path deviation were the primary measurement types; however, 
supporting measurements including rates, accelerations, and control positions were also recorded. All 
data analyses were performed on parameters sampled at five samples per second. 


Manifold Pressure (in inches of mercury) 
Side Stick (pitch in inches; roll in degrees) 
Throttle (in inches) 

Rudder Position (in degrees) 

Flaps (in degrees) 

Global Positioning System (GPS) Time 
Latitude (in degrees, minutes, and seconds) 
Longitude (in degrees, minutes, and 
seconds) 

Magnetic Heading (in degrees) 

True Heading (in degrees) 

Pitch Attitude (in degrees) 

Roll Attitude (in degrees) 

Roll Rate (in degrees per second) 

Pitch Rate (in degrees per second) 

Yaw Rate (in degrees per second) 

Heading Rate (in degrees per second) 
Filtered Roll Rate (in degrees per second) 
Filtered Pitch Rate (in degrees per second) 
Filtered Yaw Rate (in degrees per second) 
Filtered Heading Rate (in degrees per 
second) 

Turn Coordinator Rate (in degrees per 
second) 

X Acceleration (in Gs) 

Y Acceleration (in Gs) 

Z Acceleration (in Gs) 

Filtered X Acceleration (in Gs) 


Filtered Y Acceleration (in Gs) 

Filtered Z Acceleration (in Gs) 

Indicated Airspeed (in knots) 

Calibrated Airspeed (in knots) 

True Airspeed (in knots) 

Filtered Indicated Airspeed (in knots) 
Filtered True Airspeed (in knots) 

Vertical Speed (in feet per minute) 

Filtered Vertical Speed (in feet per minute) 
Ground Speed (in knots) 

Pressure Altitude (in feet) 

Filtered Pressure Altitude (in feet) 

True Altitude (in feet) 

Filtered True Altitude (in feet) 

Outside Air Temperature (in degrees 
Celsius) 

Track Angle (in degrees) 

Elevator (in degrees) 

Destination Waypoint 

Distance to Waypoint 

Lateral Path Deviation (in nautical miles) 

Track (in degrees) 

Desired Track (in degrees) 

Groundspeed (in knots) 

Bearing to Destination (in degrees) 

Discrete signals from each event marker in 
the cabin 
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APPENDIX B. Correction Equations Used for the Air Data, Attitude and 
Heading Reference System (ADAHRS) and Data Acquistion System (DAS) 


The Evaluation Pilots (EPs) used standard flight instrumentation to control airspeed, altitude, and 
vertical speed during the Small Aircraft Transportation System (SATS) Higher Volume Operations 
(HVO) Self-Separation and Sequencing (SSS) Flight Experiment. This instrumentation included an 
altimeter, airspeed indicator, and vertical speed indicator, all of which are mechanical, panel-mounted 
instruments that measure air pressures within the pitot-static system to provide analog outputs. None of 
these instruments provided digital information that was needed by the research software or for recording 
in the data acquisition system (DAS). However, this information was available in digital formats from 
the Air Data, Attitude, and Heading Reference System (ADAHRS) that is part of the baseline research 
system on the test aircraft. 


It was necessary to correct the digital data from the ADAHRS to match similar analog data provided 
to the EP by the standard flight instrumentation. Data to develop the equations used for these corrections 
were attained during ground tests by simulating specific pressure changes to the pitot-static system on the 
aircraft. The aircraft’s instruments were “tapped” before each indication was recorded to account for in- 
flight vibration. Airspeed measurements were taken in increments of 10 knots (kts), from 80 to 130 kts; 
altitude measurements were taken in increments of 100 feet (ft.), from 1900 to 6100 ft.; and vertical speed 
measurements were taken in increments of 100 feet per minute (fpm), from -1200 to 500 fpm. All of 
these data were recorded in both ascending and descending order. 


A linear correlation was made between the ADAHRS data and the standard flight instrumentation 
data to derive the following correction equations for vertical speed, airspeed, and altitude provided by the 
ADAHRS. These equations were used by the research software on a real-time basis. 


Vertical speed 

Y = 1.0442X+ 1.4555 

fpm 

Airspeed 

Y = 0.8345X + 28.715 

kts. 

Altitude 

Y = 0.995 IX- 81.249 

ft. 


Where: 

X is uncorrected data 
Y corrected data 
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APPENDIX C. Details Regarding Flight Path Parameter Deviation Data 
Extraction and Reduction 


The flight path deviation data that were analyzed to assess the Evaluation Pilot’s (EP) ability to fly 
the global positioning system (GPS) approaches with the Small Aircraft Transportation System (SATS) 
Higher Volume Operations (HVO) procedures were flight parameters that were to remain constant or 
nulled during flight. The EPs were instructed to null lateral path deviation and maintain constant airspeed 
during all segments of the approach path and maintain constant altitude during the level-flight segments. 


Each of the test conditions contained flight paths that had multiple level-flight segments followed by 
descent segments. A methodology was developed to consistently determine when the EP had transitioned 
between level-flight and descending path segments with enough time for the altitude and vertical speeds 
to become stable for their appropriate segments. Once the start and stop times for each path segment for 
each test run was established, the data could be further sectioned for the calculation of root mean squared 
error (RMSE) values. 


The following exponential moving average (EMA) of altitude was applied to establish a consistent 
method of determining flight segments: 


(Cl) Exponential Moving Average = (Current Data Value x (Exponential Percentage)) + (previous 
exponential moving average value x ( 1 - (Exponential Percentage)) 


The relationship of the moving average calculation of the altitude parameter was compared to the 
value of the parameter itself. In an EMA, more weight is given to more recent data. The greater weight 
helped determine (within a shorter period) an intent by the EP to either maintain altitude, initiate a 
descent, or level off. The flight path parameter deviation data were requested at a rate of five samples per 
second (sps). EMA was calculated over a period of 5 seconds, or 25 samples. 

2 2 

(C2) Exponential Percentage = = = 0.077 

Samples + 1 25 + 1 

Substituting the above value, the EMA formula is as follows: 

(C3) Exponential Moving Average = 

(Current Data Value x 0.077)+ (previous exponential moving average value )x (1-0.077) 


For the first calculation of the EMA, the current data value of altitude was used as input for the 
“previous EMA value.” Subsequently, this first calculation of EMA became the value for the “previous 
EMA value,” and the EMA calculation was performed on all subsequent altitude data. Ideally, when the 
EMA minus the current altitude is less than zero (i.e., EMA - current altitude < 0), this would signal the 
beginning of a descent. However, “whiplash” was observed where the difference between the EMA and 
the current altitude data value became negative and shortly thereafter became positive again (i.e., 
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representing a false indication). To reduce the false indications, a value of -0.5 was used. When the 
value of “EMA - current altitude” approached -0.5 from above, false indications were greatly reduced. 
When the value of “EMA - current altitude” approached the value -0.5 from below, the EP was leveling 
off from a descent. Using the EMA method of segmenting data, the flight segments for the beginning of 
descent and the beginning of level off could be consistently determined with minimum subjective 
judgment required by the experimenters. 


To eliminate each EP’s transition from level-flight to a descent and vice versa, only the middle 30 
seconds of descent between the start of descent and level off were analyzed. This helped ensure that the 
data that were analyzed corresponded to the point at which the EP reached, and was attempting to 
maintain, a constant rate of descent. 


For each flight segment, root mean squared error (RMSE) values were determined by using the 
corrected data and measuring their deviation from a “target value.” Target values were defined from 
instrument approach procedures and included altitudes, airspeeds, and lateral path deviations. Given the 
deviation from the targeted value, the RMSE values associated with altitude and airspeed could be 
calculated for every straight-and-level segment; airspeed RMSE values could be calculated for all descent 
segments; and lateral path deviation RMSE values could be calculated for all segments except those 
involving holding patterns. The equation used to calculated RMSE values was as follows: 


(C4) 


RMSE = 


N 

^ (ParameterActual - ParameterTarget) 2 



N 


In this equation, “parameter actual” is represented by the corrected raw data, and “N” is the number 
of data points used in the flight segment RMSE calculation. Target values were depicted by the 
instrument approach charts provided to the EPs prior to the experiment flight. The target values for each 
parameter depended upon the flight segment definition such that: 


• The airspeed target was 120 knots (kts) prior to the Final Approach Fix (FAF) and was applied to 
all flight segments; and 


• Altitude targets varied from 5,500 feet (ft.) to 2,420 ft. and were applied to only the straight-and- 
level defined flight segments in each test condition. 

Once the RMSE value was calculated for each applicable parameter in each flight segment, a total 
average value was calculated for each parameter recorded during a given replicate of a given test 
condition for a given EP. Table Cl shows a sample test condition RMSE table. 
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Table Cl. Root Mean Squared Error (RMSE) Values Collected from an Evaluation Pilot during the First Replicate 
of the Straight In, In-Trail Approach using the Method of Approach Separation 2 Display Format 


Flight Segment 

Airspeed 

Altitude 

Lateral Path deviation 


(in knots) 

(in feet) 

(in nautical miles) 

Level 5500 

1.01 

42.97 

0.09 

Descent to 4500 

4.37 

Not Applicable 

0.13 

Level 4500 

3.21 

25.48 

0.09 

Descent to 3700 

0.92 

Not Applicable 

0.17 

Level 3700 

3.00 

15.75 

0.16 

Average RMSE: 

2.50 

28.07 

0.13 


The average RMSE values (Table Cl) were analyzed by way of 2 (Approach Type) x 3 (Display Type) x 
2 (Replicate) Analysis of Variance (ANOVA) tests. 


Adherence to the Federal Aviation Administration’s (FAA) Instrument Rating Practical Test 
Standards (PTS) was determined using the same target values used in calculating the RMSE values for 
each flight path parameter [Cl], These target values represented the “correct” instrumentation reading 
that the EP was instructed to “fly to” during the test conditions’ different flight segments. The number 
and duration of each failure to adhere to the PTS associated with each flight path parameter was 
determined for each flight segment completed by each EP during each replicate of each test condition. 
Then, for a given flight path parameter, the total time associated with failures to fly within the PTS 
performance criteria was divided by the total time flying to obtain a percentage of time spent flying out of 
PTS conformance. These percentage values were analyzed by way of 2 (Approach Type) x 3 (Display 
Type) x 2 (Replicate) Analysis of Variance (ANOVA) tests. 

Reference 


[Cl] Federal Aviation Administration: Instrument Rating Practical Test Standards for Airplane. FAA- 
S-8081-4C with Changes 1 & 2, U.S. Department of Transportation (Washington, D.C.), 1998. 
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APPENDIX D. Detailed Results of Flight Path Parameter Deviation Data 

Analyses 


Deviations from assigned altitude, airspeed, and the lateral path deviation from the assigned route on 
the instrument approach were used to quantify Evaluation Pilots’ (EPs) flight performance. Root mean 
squared error (RMSE) values were calculated for these parameters for each test run. Then, a series of 2 
(Approach Type) x 3 (Display Type) x 2 (Replicate) Analysis of Variance (ANOVA) tests was conducted 
on the RMSE values of altitude, airspeed, and lateral path deviations to determine if significant 
differences existed in these values when a given display type was used during a given approach type [D 1 , 
D2, D3], 


The main effects of approach type and display type and the Approach Type x Display Type 
interaction effect were of primary interest. Means and standard deviations associated with the main effect 
of replicate and interaction effects involving the replicate factor are not presented unless they were found 
to be significant. Since the main effect of subjects is usually significant in these types of analyses, it is 
not specifically mentioned unless it was found to be not significant. For all statistical tests, a 5-percent 
significance level was set a priori. 

Altitude Deviation 


The 2x3x2 ANOVA conducted on the RMSE values of altitude revealed that: 


• No significant difference was found to exist between the altitude RMSE values associated with 
each approach type (F [1, 5] = 0.004; p = 0.954). 


• No significant difference was found to exist among the altitude RMSE values associated with the 
use of different display types (F [2, 10] = 3.30; p = 0.079). 

• A significant difference was found to exist between the altitude RMSE values associated with 
each replicate (F [1, 5] = 7.44; p = 0.041). The occurrence of smaller altitude deviations when 
EPs completed the second replicate is most likely a result of a practice or learning effect since 
EPs always performed replicate #2 after completing replicate #1 . 

• No significant Approach Type x Display Type interaction was found to exist (F [2, 10] = 0.56; p 
= 0.591). 


• No significant Approach Type x Replicate interaction was found to exist (F [1, 5] = 0.17; p = 
0.698). 


• No significant Display Type x Replicate interaction was found to exist (F [2, 10] = 0.23; p = 
0.798). 
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• No significant Approach Type x Display Type x Replicate interaction was found to exist (F [2, 
10] = 1.07; p = 0.378). 

Approach Type 

Table D1 contains the means and standard deviations associated with the altitude deviation data 
[RMSE values in feet (ft.)] attained for straight-in, in-trail approaches and simultaneous arrival 
approaches. Sample size equals 36 since altitude deviations were averaged across the three display types. 


Table Dl. Root Mean Squared Error Altitude Deviations Associated with Approach Type 


Approach Type 

Mean (ft.) 

Standard Deviation (ft.) 

Sample Size 

Sttaight In, In-ttail 

39.40 

13.99 

36 

Simultaneous Arrival 

39.52 

12.46 

36 


EPs maintained assigned altitudes equally well when they performed straight-in, in-trail approaches and 
when they performed simultaneous arrival approaches. 

Display Type 

Table D2 contains the means and standard deviations associated with the altitude deviation data 
(RMSE values in ft.) attained for the Baseline display format, the MAS 1 display format, and the MAS 2 
display format. Sample size equals 24 since altitude deviations were averaged across the two approach 
types. 


Table D2. Root Mean Squared Error Altitude Deviations Associated with Display Type 


Display Type 

Mean (ft.) 

Standard Deviation (ft.) 

Sample Size 

Baseline 

37.53 

11.76 

24 

Method of Approach Separation 1 

37.88 

8.95 

24 

Method of Approach Separation 2 

42.98 

10.25 

24 


EPs maintained assigned altitudes equally well when they performed approaches using the Baseline 
display format, the MAS 1 display format, and the MAS 2 display format. 

Replicate 


Table D3 contains the means and standard deviations associated with the altitude deviation data 
(RMSE values in ft.) attained for replicate #1 and replicate #2. Sample size equals 36 since altitude 
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deviations were collected from six EPs, each of whom completed a series of six test conditions twice. 


Table D3. Root Mean Squared Error Altitude Deviations Associated with Display Type 


Replicate 

Mean (ft.) 

Standard Deviation (ft.) 

Sample Size 

#1 

43.57 

13.32 

36 

#2 

35.35 

11.78 

36 


As noted earlier, the ANOVA revealed that the main effect of replicate was statistically significant. 
An examination of the mean altitude deviations associated with each replicate shows that smaller altitude 
deviations occurred when EPs completed the second replicate. This finding is most likely a result of a 
practice or learning effect since EPs always performed replicate #2 after completing replicate #1. 


Approach Type x Display Type 


Table D4 and Figure D1 contain the means and standard deviations associated with the altitude 
deviation data (RMSE values in ft.) attained when straight-in, in-trail approaches and simultaneous arrival 
approaches were performed using different display types. Sample size equals 12 since altitude deviations 
were not averaged across approach type or display type. 

Table D4. Root Mean Squared Error Altitude Deviations Associated with Approach Type x Display Type 


Test Condition 

Mean (ft.) 

Standard Deviation (ft.) 

Sample Size 

Straight In, In-trail Approach / 
Baseline Display Format 

36.82 

13.69 

12 

Simultaneous Arrival Approach / 
Baseline Display Format 

38.24 

14.41 

12 

Straight In, In-trail Approach / 

Method of Approach Separation 1 Display Format 

36.64 

11.91 

12 

Simultaneous Arrival Approach / 

Method of Approach Separation 1 Display Format 

39.11 

11.13 

12 

Straight In, In-trail Approach / 

Method of Approach Separation 2 Display Format 

44.75 

15.70 

12 

Simultaneous Arrival Approach / 

Method of Approach Separation 2 Display Format 

41.20 

12.54 

12 
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Straight In Sim. Arrival Straight In Sim. Arrival Straight In Sim. Arrival 
Baseline Baseline MAS 1 MAS 1 MAS 2 MAS 2 


Approach Type x Display Type 

Figure Dl. Mean altitude deviations associated with Approach Type x Display Type. 

EPs maintained assigned altitudes equally well when they performed different approach types using 
different types of displays. 

Airspeed Deviation 


The 2x3x2 ANOVA conducted on the RMSE values of airspeed revealed that: 


• A significant difference was found to exist between the airspeed RMSE values associated with 
each approach type (F [1, 5] = 6.80; p = 0.048). 


• No significant difference was found to exist among the airspeed RMSE values associated with the 
use of different display types (F [2, 10] = 0.49; p = 0.627). 


• No significant difference was found to exist between the airspeed RMSE values associated with 
each replicate (F [1, 5] = 0.90; p = 0.386). 


• No significant Approach Type x Display Type interaction was found to exist (F [2, 10] = 0.55; p 
= 0.595). 
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• No significant Approach Type x Replicate interaction was found to exist (F [1, 5] = 0.01; p = 
0.939). 


• No significant Display Type x Replicate interaction was found to exist (F [2, 10] = 0.13; p = 
0.883). 


• No significant Approach Type x Display Type x Replicate interaction was found to exist (F [2, 
10] = 0.28; p = 0.763). 


Approach Type 


Table D5 contains the means and standard deviations associated with the airspeed deviation data 
[RMSE values in knots (kts)] attained for straight-in, in-trail approaches and simultaneous arrival 
approaches. Sample size equals 36 since airspeed deviations were averaged across the three display 
types. 


Table D5. Root Mean Squared Error Airspeed Deviations Associated with Approach Type 


Approach Type 

Mean (kts) 

Standard Deviation (kts) 

Sample Size 

Straight In, In-trail 

3.11 

1.34 

36 

Simultaneous Arrival 

2.85 

0.99 

36 


Since the ANOVA revealed that the main effect of approach type was statistically significant, a simple 
examination of the mean airspeed deviations associated with each approach type was used to determine if 
greater airspeed deviations occurred during the straight-in, in-trail approaches or during the simultaneous 
arrival approaches. When airspeed RMSE values were averaged across the three display types, the mean 
airspeed deviation was 2.85 kts during the simultaneous arrival approaches, and the mean airspeed 
deviation was 3.1 1 kts during the straight-in, in-trail approaches. Statistically, EPs were able to maintain 
assigned airspeeds more accurately during the simultaneous arrival approaches than during the straight-in, 
in-trail approaches. However, a mean airspeed difference of just one-quarter of a knot is operationally 
insignificant. 


Display Type 


Table D6 contains the means and standard deviations associated with the airspeed deviation data 
(RMSE values in kts) attained for the Baseline display format, the MAS 1 display format, and the MAS 2 
display format. Sample size equals 24 since airspeed deviations were averaged across the two approach 
types. 
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Table D6. Root Mean Squared Error Airspeed Deviations Associated with Display Type 


Display Type 

Mean (kts) 

Standard Deviation (kts) 

Sample Size 

Baseline 

2.92 

1.20 

24 

Method of Approach Separation 1 

3.01 

1.13 

24 

Method of Approach Separation 2 

3.01 

1.24 

24 


EPs maintained assigned airspeed equally well when they performed approaches using the Baseline 
display format, the MAS 1 display format, and the MAS 2 display format. 


Approach Type x Display Type 


Table D7 and Figure D2 contain the means and standard deviations associated 
deviation data (RMSE values in kts) attained when straight-in, in-trail approaches 
arrival approaches were performed using different display types. Sample size equals 
deviations were not averaged across approach type or display type. 

Table D7. Airspeed Deviations (Root Mean Squared Error values in knots) Associated with Approach Type x 

Display Type 


with the airspeed 
and simultaneous 
12 since airspeed 


Test Condition 

Mean (kts) 

Standard Deviation (kts) 

Sample Size 

Straight In, In-trail Approach / 
Baseline Display Format 

2.83 

1.09 

12 

Simultaneous Arrival Approach / 
Baseline Display Format 

3.02 

1.35 

12 

Straight In, In-trail Approach / 

Method of Approach Separation 1 Display Format 

3.28 

1.48 

12 

Simultaneous Arrival Approach / 

Method of Approach Separation 1 Display Format 

2.74 

0.57 

12 

Straight In, In-trail Approach / 

Method of Approach Separation 2 Display Format 

2.81 

0.95 

12 

Simultaneous Arrival Approach / 

Method of Approach Separation 2 Display Format 

3.22 

1.48 

12 
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Figure D2. Mean airspeed deviations associated with Approach Type x Display Type. 

EPs maintained assigned airspeeds equally well when they performed different approach types using 
different types of displays. 


2.81 


Lateral Path Deviation 


The 2x3x2 ANOVA conducted on the RMSE values of lateral path deviation revealed that: 


• No significant difference was found to exist between the lateral path deviation RMSE values 
associated with each approach type (F [1, 5] = 0.94; p = 0.376). 


• No significant difference was found to exist among the lateral path deviation RMSE values 
associated with the use of different display types (F [2, 10] = 1.43; p = 0.284). 


• No significant difference was found to exist between the lateral path deviation RMSE values 
associated with each replicate (F [1, 5] = 1.49; p = 0.276). 


• No significant Approach Type x Display Type interaction was found to exist (F [2, 10] = 2.08; p 
= 0.176). 
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No significant Approach Type x Replicate interaction was found to exist (F [1, 5] = 1.08; p = 
0.345). 


• No significant Display Type x Replicate interaction was found to exist (F [2, 10] = 0.90; p = 
0.435). 


• No significant Approach Type x Display Type x Replicate interaction was found to exist (F [2, 
10] = 0.55; p = 0.591). 


Approach Type 


Table D8 contains the means and standard deviations associated with the lateral path deviation data 
[RMSE values in nautical miles (n. m.)] attained for straight-in, in-trail approaches and simultaneous 
arrival approaches. Sample size equals 36 since lateral path deviations were averaged across the three 
display types. 

Table D8. Root Mean Squared Error Lateral Path Deviations Associated with Approach Type 


Approach Type 

Mean (n. m.) 

Standard Deviation (n. m.) 

Sample Size 

Straight In, In-trail 

0.08 

0.04 

36 

Simultaneous Arrival 

0.07 

0.05 

36 


EPs maintained the assigned course equally well when they performed straight-in, in-trail approaches and 
when they performed simultaneous arrival approaches. 


Display Type 


Table D9 contains the means and standard deviations associated with the lateral path deviation data 
(RMSE values in n. m.) attained for the Baseline display format, the MAS 1 display format, and the MAS 
2 display format. Sample size equals 24 since lateral path deviations were averaged across the two 
approach types. 


Table D9. Root Mean Squared Error Lateral Path Deviations Associated with Display Type 


Display Type 

Mean (n. m.) 

Standard Deviation (n. m.) 

Sample Size 

Baseline 

0.07 

0.05 

24 

Method of Approach Separation 1 

0.09 

0.05 

24 

Method of Approach Separation 2 

0.07 

0.03 

24 
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EPs maintained assigned course equally well when they performed approaches using the Baseline display 
format, the MAS 1 display format, and the MAS 2 display format. 


Approach Type x Display Type 


Table DIO and Figure D3 contain the means and standard deviations associated with the lateral path 
deviation data (RMSE values in n. m.) attained when straight-in, in-trail approaches and simultaneous 
arrival approaches were performed using different display types. Sample size equals 12 since lateral path 
deviations were not averaged across approach type or display type. 

Table DIO. Root Mean Squared Error Lateral Path Deviations Associated with Approach Type x Display Type 


Test Condition 

Mean (n. m.) 

Standard Deviation (n. m.) 

Sample 

Size 

Straight In, In-trail Approach / 
Baseline Display Format 

0.07 

0.03 

12 

Simultaneous Arrival Approach / 
Baseline Display Format 

0.08 

0.06 

12 

Straight In, In-trail Approach / 

Method of Approach Separation 1 Display Format 

0.10 

0.05 

12 

Simultaneous Arrival Approach / 

Method of Approach Separation 1 Display Format 

0.07 

0.05 

12 

Straight In, In-trail Approach / 

Method of Approach Separation 2 Display Format 

0.07 

0.03 

12 

Simultaneous Arrival Approach / 

Method of Approach Separation 2 Display Format 

0.07 

0.03 

12 
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Approach Type x Display Type 

Figure D3. Mean lateral path deviations associated with Approach Type x Display Type. 

EPs maintained assigned course equally well when they performed different approach types using 
different types of displays. 
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APPENDIX E. Detailed Results of Adherence to PTS for the Instrument 
Rating Data Analyses 


Percentages of time that Evaluation Pilots (EPs) failed to adhere to the Practical Test Standards (PTS) 
for the Instrument Rating were used to quantify flight performance. The duration (i.e., elapsed time in 
seconds) of each failure to fly within the PTS was the difference between the point in time when a PTS 
parameter was breached and the point in time when adherence to that PTS parameter was regained. For a 
given flight path parameter, the total time associated with failures to fly within PTS performance criteria 
was divided by the total time flying to obtain a percentage of time spent flying out of PTS conformance. 


Percentages of time that EPs failed to adhere to the PTS were calculated for altitude, airspeed, and 
lateral path deviation. Data used to assess PTS conformance were obtained from 30-second segments 
during descents where the parameters would be stabilized. Airspeed and lateral path deviation data were 
assessed during both level-flight and descents. Altitude data were assessed only during level-flight 
segments. All EPs successfully adhered to the PTS performance criteria for lateral tracking for the 
Instrument Rating (i.e., ± 1.0 n. m. prior to the FAF, and ± 0.225 n. m. after the FAF) 100-percent of the 
time. Therefore, 2 (Approach Type) x 3 (Display Type) x 2 (Replicate) ANOVAs were conducted on the 
percentages of time that EPs failed to adhere to the PTS for altitude and airspeed to determine if 
significant differences existed in these values when a given display type was used during a given 
approach type [El, E2, E3]. 


The main effects of approach type and display type and the Approach Type x Display Type 
interaction effect were of primary interest. Means and standard deviations associated with the main effect 
of replicate and interaction effects involving the replicate factor are not presented unless they were found 
to be significant. Since the main effect of subjects is usually significant in these types of analyses, it is 
not specifically mentioned unless it was found to be not significant. For all statistical tests, a 5-percent 
significance level was set a priori. 

Adherence to Altitude PTS 


The 2x3x2 ANOVA conducted on the percentages of time that EPs failed to adhere to the PTS for 
altitude revealed that: 


• No significant difference was found to exist among the percentages of altitude PTS 

nonconformance associated with subjects (F [1, 5] = 6.18; p = 0.055). 

• No significant difference was found to exist between the percentages of altitude PTS 

nonconformance associated with each approach type (F [1, 5] = 3.46; p = 0.122). 

• No significant difference was found to exist among the percentages of altitude PTS 

nonconformance associated with the use of different display types (F [2, 10] = 0.09; p = 0.918). 


54 



• No significant difference was found to exist between the percentages of altitude PTS 
nonconformance associated with each replicate (F [1, 5] = 2.44; p = 0.179). 

• No significant Approach Type x Display Type interaction was found to exist (F [2, 10] = 1.29; p 
= 0.317). 


• No significant Approach Type x Replicate interaction was found to exist (F [1, 5] = 0.18; p = 
0.687). 


• No significant Display Type x Replicate interaction was found to exist (F [2, 10] = 0.55; p = 
0.592). 


• No significant Approach Type x Display Type x Replicate interaction was found to exist (F [2, 
10] = 1.17; p = 0.350). 

Subjects 

Across EPs, there was very little variability among the percentages of altitude PTS nonconformance. 
EPs adhered to the altitude PTS in a very consistent manner with each other across the test conditions. 


Approach Type 


Table El contains the means and standard deviations associated with the percentages of altitude PTS 
nonconformance attained for straight-in, in-trail approaches and simultaneous arrival approaches. Sample 
size equals 36 since percentages were averaged across the three display types. 


Table El. Percentages of Altitude PTS Nonconformance Associated with Approach Type 


Approach Type 

Mean 

Standard Deviation 

Sample Size 

Straight In, In-trail 

1.08 

2.71 

36 

Simultaneous Arrival 

2.01 

3.23 

36 


EPs adhered to the altitude PTS equally well when they performed straight-in, in-trail approaches and 
when they performed simultaneous arrival approaches. 


Display Type 


Table E2 contains the means and standard deviations associated with the percentages of altitude PTS 
nonconformance attained for the Baseline display format, the MAS 1 display format, and the MAS 2 
display format. Sample size equals 24 since percentages were averaged across the two approach types. 
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Table E2. Percentages of Altitude PTS Nonconformance Associated with Display Type 


Display Type 

Mean 

Standard Deviation 

Sample Size 

Baseline 

1.52 

3.48 

24 

Method of Approach Separation 1 

1.68 

2.28 

24 

Method of Approach Separation 2 

1.43 

3.23 

24 


EPs adhered to the altitude PTS equally well when they performed approaches using the Baseline display 
format, the MAS 1 display format, and the MAS 2 display format. 


Approach Type x Display Type 


Table E3 and Figure El contain the means and standard deviations associated with the percentages of 
altitude PTS nonconformance attained when straight-in, in-trail approaches and simultaneous arrival 
approaches were performed using different display types. Sample size equals 12 since percentages were 
not averaged across approach type or display type. 

Table E3. Percentages of Altitude PTS Nonconformance Associated with Approach Type x Display Type 


Test Condition 

Mean 

Standard 

Deviation 

Sample Size 

Straight In, In-trail Approach / 
Baseline Display Format 

0.75 

1.86 

12 

Simultaneous Arrival Approach / 
Baseline Display Format 

2.30 

4.53 

12 

Straight In, In-trail Approach / 

Method of Approach Separation 1 Display Format 

0.71 

1.47 

12 

Simultaneous Arrival Approach / 

Method of Approach Separation 1 Display Format 

2.65 

2.57 

12 

Straight In, In-trail Approach / 

Method of Approach Separation 2 Display Format 

1.77 

4.11 

12 

Simultaneous Arrival Approach / 

Method of Approach Separation 2 Display Format 

1.09 

2.16 

12 
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Figure El. Mean percentages of altitude Practical Test Standard (PTS) nonconformance associated with Approach 

Type x Display Type. 

EPs adhered to the altitude PTS equally well when they performed different approach types using 
different types of displays. 

Adherence to Airspeed PTS 


The 2x3x2 ANOVA conducted on the percentages of time that EPs failed to adhere to the PTS for 
airspeed revealed that: 


• No significant difference was found to exist among the percentages of airspeed PTS 

nonconformance associated with subjects (F [1, 5] = 2.38; p = 0.183). 

• No significant difference was found to exist between the percentages of airspeed PTS 

nonconformance associated with each approach type (F [1, 5] = 2.35; p = 0.186). 

• No significant difference was found to exist among the percentages of airspeed PTS 

nonconformance associated with the use of different display types (F [2, 10] = 2.38; p = 0.142). 
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• No significant difference was found to exist between the percentages of airspeed PTS 
nonconformance associated with each replicate (F [1, 5] = 0.35; p = 0.580). 

• No significant Approach Type x Display Type interaction was found to exist (F [2, 10] = 1.72; p 
= 0.228). 


• No significant Approach Type x Replicate interaction was found to exist (F [1, 5] = 0.001; p = 
0.980). 


• No significant Display Type x Replicate interaction was found to exist (F [2, 10] = 0.17; p = 
0.849). 


• No significant Approach Type x Display Type x Replicate interaction was found to exist (F [2, 
10] = 0.15; p = 0.861). 


Subjects 


Across EPs, there was very little variability among the percentages of airspeed PTS nonconformance. 
EPs adhered to the airspeed PTS in a very consistent manner with each other across the test conditions. 


Approach Type 

Table E4 contains the means and standard deviations associated with the percentages of airspeed PTS 
nonconformance attained for straight-in, in-trail approaches and simultaneous arrival approaches. Sample 
size equals 36 since percentages were averaged across the three display types. 


Table E4. Percentages of Airspeed PTS Nonconformance Associated with Approach Type 


Approach Type 

Mean 

Standard Deviation 

Sample Size 

Straight In, In-trail 

1.11 

3.72 

36 

Simultaneous Arrival 

0.30 

1.37 

36 


EPs adhered to the airspeed PTS equally well when they performed straight-in, in-trail approaches and 
when they performed simultaneous arrival approaches. 


Display Type 


Table E5 contains the means and standard deviations associated with the percentages of airspeed PTS 
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nonconformance attained for the Baseline display format, the MAS 1 display format, and the MAS 2 
display format. Sample size equals 24 since percentages were averaged across the two approach types. 


Table E5. Percentages of Airspeed PTS Nonconformance Associated with Display Type 


Display Type 

Mean 

Standard Deviation 

Sample Size 

Baseline 

0.48 

1.66 

24 

Method of Approach Separation 1 

0.25 

0.90 

24 

Method of Approach Separation 2 

1.37 

4.49 

24 


EPs adhered to the airspeed PTS equally well when they performed approaches using the Baseline display 
format, the MAS 1 display format, and the MAS 2 display format. 

Approach Type x Display Type 

Table E6 and Figure E2 contain the means and standard deviations associated with the percentages of 
airspeed PTS nonconformance attained when straight-in, in-trail approaches and simultaneous arrival 
approaches were performed using different display types. Sample size equals 12 since percentages were 
not averaged across approach type or display type. 


Table E6. Percentages of Airspeed PTS Nonconformance Associated with Approach Type x Display Type 


Test Condition 

Mean 

Standard 

Deviation 

Sample Size 

Straight In, In-trail Approach / 
Baseline Display Format 

0.25 

0.62 

12 

Simultaneous Arrival Approach / 
Baseline Display Format 

0.72 

2.30 

12 

Straight In, In-trail Approach / 

Method of Approach Separation 1 Display Format 

0.50 

1.24 

12 

Simultaneous Arrival Approach / 

Method of Approach Separation 1 Display Format 

0.00* 

0.00* 

12 

Straight In, In-trail Approach / 

Method of Approach Separation 2 Display Format 

2.58 

6.22 

12 

Simultaneous Arrival Approach / 

Method of Approach Separation 2 Display Format 

0.17 

0.58 

12 


* NOTE: A value of zero indicates conformance 100-percent of the time. 
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Figure E2. Mean percentages of airspeed Practical Test Standards (PTS) nonconformance associated with Approach 

Type x Display Type. 

EPs adhered to the airspeed PTS equally well when they performed different approach types using 
different types of displays. 

Adherence to the PTS Performance Criteria for Lateral Tracking 


All EPs successfully adhered to the CDI PTS for the Instrument Rating 100-percent of the time 
during all six of the experiment’s test conditions. It was concluded that EPs were able to maintain their 
course with an appropriate level of precision while performing straight-in, in-trail approaches and 
simultaneous arrival approaches using the Baseline display format, the MAS 1 display format, and the 
MAS 2 display format. 
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APPENDIX F. Detailed Results of Subjective Workload Data Analyses 


Evaluation Pilots (EPs) used the Air Force Flight Test Center’s (AFFTC) Seven-Point Subjective 
Workload Estimate Scale and the Modified Cooper-Harper (MCH) Rating Scale to rate the level of 
workload that they experienced during each of the experiment’s six test conditions [Gl, G2], Since each 
test condition was performed twice during the course of the experiment, each EP provided 12 AFFTC 
workload ratings and 12 MCH workload ratings. For each EP, the two AFFTC workload ratings 
associated with a given test condition were averaged together to yield a set of six mean workload ratings, 
and the two MCH workload ratings associated with a given test condition were averaged together to yield 
a set of six mean workload ratings. Nonparametric tests were employed as a conservative method for 
analyzing workload ratings associated with discrete rating scale items. For all statistical tests, a 5-percent 
significance level was set a priori. 

Approach Type 


AFFTC Workload Ratings 


Table FI contains the means and standard deviations for the AFFTC workload ratings associated with 
performing straight-in, in-trail approaches and simultaneous arrival approaches. Sample size equals 18 
since workload ratings were averaged across the two replicates and the three display types. 

Table FI. Air Force Flight Test Center Workload Estimate Scale Ratings Associated with Approach Type 


Approach Type 

Mean 

Standard Deviation 

Sample Size 

Straight In, In-trail 

2.72 

0.52 

18 

Simultaneous Arrival 

2.61 

0.53 

18 


A Wilcoxon Test (i.e., a nonparametric within-subject test appropriate for analyzing two related samples 
of ordinal data) performed on these means revealed that EPs reported experiencing equivalent levels of 
workload during the straight-in, in-trail approaches and the simultaneous arrival approaches (p = 0.6546) 
[G3], 


MCH Workload Ratings 


Table F2 contains the means and standard deviations of with the MCH workload ratings associated 
with performing straight-in, in-trail approaches and simultaneous arrival approaches. Sample size equals 
18 since workload ratings were averaged across the two replicates and the three display types. 
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Table F2. Modified Cooper-Harper Workload Ratings Associated with Approach Type 


Approach Type 

Mean 

Standard Deviation 

Sample Size 

Straight In, In-trail 

2.42 

0.97 

18 

Simultaneous Arrival 

2.14 

1.03 

18 


A Wilcoxon Test revealed that EPs reported experiencing equivalent levels of workload during the 
straight-in, in-trail approaches and the simultaneous arrival approaches (p = 0.0833). 

Display Type 

AFFTC Workload Ratings 

Table F3 contains the means and standard deviations for the AFFTC workload ratings associated with 
performing approaches using the Baseline display format, the MAS 1 display format, and the MAS 2 
display format. Sample size equals 12 since workload ratings were averaged across the two replicates and 
the two approach types. 


Table F3. Air Force Flight Test Center Workload Estimate Scale Ratings Associated with Display Type 


Display Type 

Mean 

Standard Deviation 

Sample Size 

Baseline 

2.63 

0.53 

12 

Method of Approach Separation 1 

2.63 

0.57 

12 

Method of Approach Separation 2 

2.75 

0.50 

12 


A Friedman Test (i.e., a nonparametric within-subject test appropriate for analyzing three or more related 
samples of ordinal data) performed on these means revealed that EPs reported experiencing equivalent 
AFFTC workload ratings when different display types were used (X 2 [2] = 2.2353; p = 0.3270) [G3, G4]. 

MCH Workload Ratings 


Table F4 contains the means and standard deviations for the MCH workload ratings associated with 
performing approaches using the Baseline display format, the MAS 1 display format, and the MAS 2 
display format. Sample size equals 12 since workload ratings were averaged across the two replicates and 
the two approach types. 
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Table F4. Modified Cooper-Harper Workload Ratings Associated with Display Type 


Display Type 

Mean 

Standard Deviation 

Sample Size 

Baseline 

2.13 

0.98 

12 

Method of Approach Separation 1 

2.38 

1.11 

12 

Method of Approach Separation 2 

2.33 

0.96 

12 


A Friedman Test performed on these means revealed that EPs reported experiencing equivalent MCH 
workload ratings when different display types were used (X 2 [2] = 0.7778; p = 0.6778). 

Approach Type x Display Type 
AFFTC Workload Ratings 


Table F5 and Figure FI contain the means and standard deviations for the AFFTC workload ratings 
associated with performing straight-in, in-trail approaches and simultaneous arrival approaches using 
different display types. Sample size equals 6 since workload ratings were averaged across replicates. 

Table F5. Air Force Flight Test Center Workload Estimate Scale Ratings Associated with Approach Type x Display 

Type 


Test Condition 

Mean 

Standard Deviation 

Sample Size 

Straight In, In-trail Approach / 
Baseline Display Format 

2.58 

0.58 

6 

Simultaneous Arrival Approach / 
Baseline Display Format 

2.67 

0.52 

6 

Straight In, In-trail Approach / 

Method of Approach Separation 1 Display Format 

2.75 

0.52 

6 

Simultaneous Arrival Approach / 

Method of Approach Separation 1 Display Format 

2.50 

0.63 

6 

Straight In, In-trail Approach / 

Method of Approach Separation 2 Display Format 

2.83 

0.52 

6 

Simultaneous Arrival Approach / 

Method of Approach Separation 2 Display Format 

2.67 

0.52 

6 
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Table F6 and Figure F2 contain the means and standard deviations for the MCH workload ratings 
associated with performing straight-in, in-trail approaches and simultaneous arrival approaches using 
different display types. Sample size equals 6 since workload ratings were averaged across replicates. 
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Table F6. Modified Cooper-Harper Workload Ratings Associated with Approach Type x Display Type 


Test Condition 

Mean 

Standard Deviation 

Sample Size 

Straight In, In-trail Approach / 
Baseline Display Format 

2.08 

1.07 

6 

Simultaneous Arrival Approach / 
Baseline Display Format 

2.17 

0.98 

6 

Straight In, In-trail Approach / 

Method of Approach Separation 1 Display Format 

2.58 

0.97 

6 

Simultaneous Arrival Approach / 

Method of Approach Separation 1 Display Format 

2.17 

1.29 

6 

Straight In, In-trail Approach / 

Method of Approach Separation 2 Display Format 

2.58 

0.97 

6 

Simultaneous Arrival Approach / 

Method of Approach Separation 2 Display Format 

2.08 

0.97 

6 
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Figure F2. Mean Modified Cooper-Harper (MCH) workload ratings associated with Approach Type x Display Type. 
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A Friedman Test performed on these means revealed that equivalent workload ratings were reported to be 
experienced when different display types were used during the straight-in, in-trail approaches and the 
simultaneous arrival approaches (X [5] = 5.0000; p = 0.4159). 
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